Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create /top-helper-ai slash command #1163

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

surajkumar
Copy link
Contributor

@surajkumar surajkumar commented Sep 13, 2024

Description

This PR introduces a new slash command /top-helper-ai. The idea of this command is to use ChatGPT to determine who the top helpers of the current month are.

The command works like this:

  1. Loop over all the questions in the #questions forum channel
  2. Break if the question was not created since the start of the current month
  3. Fetch all the messages that were sent in a given question
  4. Send message to ChatGPT asking who the helper was in that given question
  5. Fetch the users that appeared as the "top helper"
  6. Respond with an embed showing the results

This PR is the first take. In a thread that has multiple top helpers, only 1 of them will be the "top" helper. We'll see how it goes. Should this work very well, the next step would be to enchance it to detect multiple helpers in a given question.

Why even create this command?

The TJ moderators currently determine the top helper based on message length. The frequent messaging (or long length messages) the more likely you are to be the top helper. This includes "shit posting" and if you're a recognised member (say, you're fairly active), the team might confuse your shit post with actual helpful content.

Top helper is a role given to people that provide quality help in the questions forum. As such, we often reward helpers with Intellij licenses and other incentives so it's pretty crucial that we are awarding such things to users that actually post helpful responses to questions.

Using ChatGPT is an attempt to better this process to keep the questions forum a helpful space.

The prompt sent to ChatGPT

The prompt sent to ChatGPT looks like this:

The following contains user IDs and their message. Using the messages provided by each user, which user ID was the most helpful/answered the question. If there are no meaningful messages, you must still choose somebody. ONLY provide the user ID of that person. Do not reply with anything else.
%s

After that prompt, the %s is replaced with all the messages in the following syntax; <userId>: <message>
A sample:

123456789: Try doing xyz
334456789: it didn't work
123456789: ok try this now, zyx

In this example, we would hope that 123456789 is the top helper.

Code exludes the OP and bots from being the top helper.

Full prompt sent:

The following contains user IDs and their message. Using the messages provided by each user, which user ID was the most helpful/answered the question. If there are no meaningful messages, you must still choose somebody. ONLY provide the user ID of that person. Do not reply with anything else.
123456789: Try doing xyz
334456789: it didn't work
123456789: ok try this now, zyx

Command output:

Embed containing:

Top helpers

user1 10
user2 5
user3 3
user4 1

The higher the number next to their name, the better the helper they are

@surajkumar surajkumar force-pushed the top-helper branch 2 times, most recently from 0d0114a to 4c5efe9 Compare September 14, 2024 17:33
@surajkumar surajkumar force-pushed the top-helper branch 2 times, most recently from 618f747 to 404f9f4 Compare September 14, 2024 18:06
@surajkumar surajkumar marked this pull request as ready for review September 14, 2024 18:10
@surajkumar surajkumar requested a review from a team as a code owner September 14, 2024 18:10
@Zabuzard
Copy link
Member

Zabuzard commented Sep 14, 2024

btw "The TJ moderators currently determine the top helper based on message count" is not true. the current command is based on message length, not count.

i havent seen anyone yet who attempted to influence it with ill intention. no one was exploiting it yet.
the only bias it really has is favoring posts with code, as they obviously tend to have a lot more characters than pure text answers.

i would be surprised though if this actually yields results really different to the ones we already have, so far the top helper lists always looked very promising.

but that just as a note, im not against the idea of using chatgpt to help. if anything, then my comments would probably rather be at how this particular implementation is using discord api to fetch a massive load of content. instead of us just saving stuff in a database or sth and fetching from there.

sth that is also interesting to check out is how this would influence our chatgpt api cost, as the amount of content send by this is quite massive. a rough estimate on the cost would be nice. but given that we are currently paying almost nothing, there is still a lot of room for using chatgpt more.

@surajkumar
Copy link
Contributor Author

Thanks @Zabuzard, around message count, that was an accident. You're right it's message length.

I'm currently evaluating the costs so I'll respond back once I can give us a rough idea around the assumption we always get mega threads so we can understand our worst case scenario.

@surajkumar surajkumar marked this pull request as draft September 15, 2024 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants